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Global  and  Basin-Scale  Ocean  Modeling  and  Prediction 


Alan  J.  Wallcraft,  Harley  E.  Hurlburt,  Robert  C.  Rhodes,  Jay  F.  Shriver* 
Naval  Research  Laboratory 
Stennis  Space  Center,  MS  39529-5004 


Abstract 

This  DoD  HPC  Challenge  project  is  part  of  a  coordinated  6. 1-6.4  effort  on  the  "Grand  Challenge" 
problem  of  eddy-resolving  global  and  basin-scale  ocean  modeling  and  prediction.  This  project  is  work 
towards  the  ultimate  goal  of  a  data  assimilative  1/32“  global  ocean  nowcast/forecast  system  including 
the  associated  basic  research  and  exploratory  development.  The  interim  goal  by  2001  is  the  transition  of 
a  1/8“  global  system  with  embedded  1/16“  Pacific  north  of  20“S  and  1/32“  subtropical  Atlantic  (9-5  TN) 
including  the  Caribbean  and  Gulf  of  Mexico.  A  data  assimilative  1/4“  global  system  was  transitioned  to 
Fleet  Numerical  Meteorology  and  Oceanography  Center  (I^MOC)  for  operational  evaluation  early  in 
FY98  and  the  1/16“  Pacific  system  is  scheduled  for  transition  by  the  end  of  FY99.  A  two  pronged 
approach  is  used  in  this  DoD  Challenge  eflfort.  The  first  is  ocean  nowcast  experiments  with  data 
assimilation  and  ocean  forecasting  experiments  initialized  from  the  experiments  with  data  assimilation, 
1/16“  Pacific  in  FY98-99  and  1/32“  subtropical  Atlantic  in  FY(X).  The  second  prong  is  "first  of  a  kind" 
ocean  model  simulations  at  the  next  higher  resolution,  1/16“  global  (completed  under  an  FY97  DoD 
HPC  Challenge  project),  1/32“  Pacific  north  of  20“S  and  1/64“  subtropicid  Atlantic  in  FY98-99 
(currently  underway  in  an  FY98-()0  DoD  HPC  Challenge  project)  and  1/32“  global  in  FYOO,  the  ultimate 
target  resolution  for  a  global  nowcast/forecast  system.  These  simulations  provide  information  on  the 
value  added  of  increased  resolution  and  input  on  tradeoffs  between  increased  resolution  and  model 
modifications  which  make  it  more  expensive  to  run.  Research  topics  include  ocean  dynamics,  model 
development  and  portability  to  different  computer  architectures,  ocean  simulation,  model  validation, 
data  assimilation,  ocean  nowcasting  and  forecasting,  observing  system  simulation,  and  applications  of 
naval  interest. 


Objectives 

This  DoD  Challenge  project  is  part  of  a  coordinated  6. 1-6.4  effort  on  the  "Grand  Challenge"  problem  of 
eddy-resolving  global  and  basin-scale  ocean  modeling  and  prediction.  The  long  term  objectives  are  to 
simulate,  understand,  nowcast  and  forecast  the  global  ocean  circulation,  and  to  increase  the  capability  to 
model  it.  Research  topics  include  ocean  dynamics,  model  development  and  portability  to  different 
computer  architectures,  ocean  simulation,  model  validation,  data  assimilation,  ocean  nowcasting  and 
forecasting,  observing  system  simulation,  and  applications  of  naval  interest.  In  addition,  the  ocean 
model  and  model  output  are  made  available  to  researchers  outside  NRL.  This  DoD  HPC  Challenge 
effort  directly  supports  5  funded  projects:  6.1  Thermodynamic  and  Topographic  Forcing  in  Global 
Ocean  Models  (ONR),  6.1  Dynamics  of  Low  Latitude  Western  Boundary  Currents  (ONR),  6.2  Basin 
Scale  Prediction  Systems  (ONR),  6.3  Scalable  Ocean  Models  with  Domain  Decomposition  and  Parallel 


Model  Components  (CHSSI),  and  6.4  Large  Scale  Ocean  Prediction  (SPAWAR).  Applications  for  the 
models  and  the  nowcast/forecast  systems  include  assimilation  and  synthesis  of  global  satellite  surface 
data;  ocean  prediction;  optimum  track  ship  routing;  search  and  rescue;  anti-submarine  warfare  and 
surveillance;  tactical  planning;  high  resolution  boundary  conditions  that  are  essential  for  even  higher 
resolution  coastal  models;  inputs  to  ice,  atmospheric  and  bio-physical  models  and  shipboard 
environmental  products;  environmental  simulation  and  synthetic  environments;  observing  system 
simulations;  ocean  research;  pollution  and  tracer  tracking  and  inputs  to  water  quality  assessment. 
Assimilation  of  satellite  altimeter  data  into  the  models  will  make  more  effective  use  of  near  realtime 
altimeter  data  from  the  Navy’s  Geosat  Follow  On  (GFO)  mission,  TOPEX/POSEEDON  and  ERS-2  via 
NAVOCEANO’s  Altimeter  Data  Fusion  Center  (ADFC).  Other  data  will  be  assimilated  as  well. 


Approach  and  Planned  Research 

Overall,  this  effort  is  work  towards  the  goal  of  developing  a  data  assimilative  1/32®  global  ocean 
nowcast/forecast  system  for  transition  to  Fleet  Numerical  Meteorology  and  Oceanography  Center 
(FNMOC).  This  includes  plans  to  participate  in  a  multi-national  Global  Ocean  Data  Assimilation 
Experiment  (GODAE)  planned  for  2003-2005,  in  addition  to  preceding  pilot  programs.  Our  effort  is 
already  represented  on  the  Scientific  Steering  Team  for  GODAE.  The  interim  goal  by  2001  is  the 
transition  of  a  1/8®  global  system  with  embedded  1/16®  Pacific  north  of  20°S  and  1/32®  subtropical 
Atlantic  (9-5  TN)  including  the  Caribbean  and  Gulf  of  Mexico.  A  data  assimilative  1/4®  globd  system 
was  transitioned  to  FNMOC  for  operational  evaluation  early  in  FY98  and  has  given  a  very  clear  picture 
showing  the  evolution  of  the  1997-98  El  Nino  (Metzger,  et  al.,  1998).  The  1/16®  Pacific  system  is 
scheduled  for  transition  by  the  end  of  FY99. 

The  Pacific  component  and  all  subsequent  systems  of  this  type  are  being  designed  for  deployment  on  a 
scalable  computer.  Since  FNMOC  does  not  yet  have  such  a  machine,  they  will  initially  be  deployed  on 
the  T3E  at  NAVO  and  later  migrated  to  a  scalable  computer  system  at  FT^OC  when  available.  These 
systems  will  serve  as  the  backbone  of  a  globally  relocatable  ocean  nowcast/forecast  capability  that  will 
address  the  Naval  need  for  littoral  or  deep  water  support  anywhere  in  the  world. 

A  two  pronged  approach  is  used  in  this  DoD  HPC  Challenge  effort.  The  first  is  ocean  nowcast 
experiments  with  data  assimilation  and  ocean  forecasting  experiments  initialized  from  the  experiments 
with  data  assimilation.  The  second  is  "first  of  a  kind"  ocean  model  simulations  at  the  next  higher 
resolution,  1/32®  Pacific  north  of  20®S,  1/64®  subtropical  Atlantic  (FY98-99)  and  1/32®  global  (FYOO). 
1/16®  global  ocean  simulations  were  completed  under  an  FY97  DoD  HPC  Challenge  Project  on  the 
CEWES  T3E.  The  "first  of  a  kind"  simulations  in  the  Atlantic  and  Pacific  are  presently  being  run  under 
an  FY98-00  DoD  HPC  Challenge  Project  on  the  T3E-900  at  NAVO.  However,  these  simulations  could 
be  run  efficiently  and  interchangeably  on  any  computer  system  configured  to  handle  simulations  this 
large  (if  it  has  a  single  global  scratch  file  system)  because  the  NRL  Layered  Ocean  Model  (NLOM), 
used  for  the  simulations,  is  scalable  and  portable  using  a  tiled  data  pardlel  programming  paradigm 
(Wallcraft  and  Moore,  1997).  This  approach  is  general  enough  that  the  technique  used  to  obtain 
scalability  on  a  given  computer  system  can  be  selected  at  compile  time  from  (1)  data  parallel,  (2)  SPMD 
message  passing,  (3)  autotasking  or  (4)  SPMD  message  passing  between  multi-processor  autotasked 
systems. 

The  simulations  within  this  project  (except  for  1/32®  global)  can  run  in  existing  large  queues  on  several 


supercomputers  within  DoD  HPC,  but  turnaround  for  large  jobs  is  historically  poor.  Through  DoD  HPC 
Challenge  resources  we  can  obtain  sufficient  turnaround  to  accomplish  the  stated  R&D  goals. 

The  first  of  a  kind  ocean  model  simulations  demonstrate  the  leading  edge  capability  to  model  the  ocean 
and  typically  are  possible  about  5  years  before  such  simulations  can  be  done  routinely.  In  addition  to 
demonstrating  future  potential  for  ocean  models,  they  are  critical  in  decision  making  when  formulating 
future  plans.  For  example,  they  provide  information  on  value  added  of  increased  resolution  and  input  on 
tradeoffs  between  increased  grid  resolution  and  model  modifications  which  make  it  more  expensive  to 
run.  NLOM  has  a  factor  of  10s  to  100s  advantage  in  computer  time  requirements  over  other  global  and 
basin-scale  ocean  models.  In  conjunction  with  a  set  of  simulations  performed  at  coarser  resolution,  these 
simulations  contribute  vital  input  in  answering  scientific  questions  about  oceanic  dynamics. 


Resolution  Requirements  for  Ocean  Modeling 

A  critical  issue  in  forecast  system  design  is  determining  the  resolution  required.  Ocean  models  require 
finer  resolution  and  more  computer  time  than  atmospheric  models  in  part  because  the  space  scales  for 
variability  due  to  flow  instabilities  (oceanic  mesosc^e  eddies  vs.  atmospheric  high  and  lows)  are  about 
20-30  times  smaller  than  found  in  the  atmosphere.  We  need  to  resolve  the  oceanic  eddy  space  scale  very 
well  because  (1)  it  is  relevant  for  most  of  the  Navy  applications  listed  earlier,  (2)  these  models  need  to 
provide  high  resolution  boundary  conditions  for  even  higher  resolution  coastal  models,  (3)  upper  ocean  - 
topographic  coupling  via  flow  instabilities  has  a  major  impact  on  the  pathways  of  many  upper  ocean 
currents  (including  mean  pathways)  and  very  fine  resolution  of  the  flow  instabilities  is  required  to  get 
sufficient  coupling  (Hurlburt,  et  al.,  1996;  Hogan  and  Hurlburt,  1997;  Hurlburt  and  Metzger,  1998),  (4) 
very  fine  resolution  is  required  to  obtain  (a)  inertial  jets  and  sharp  oceanic  fronts  which  span  major 
ocean  basins  as  observed  (Hurlburt,  et  al.,  1996)  and  (b)  the  associated  nonlinear  recirculation  gyres 
which  affect  the  shape  of  large-scale  ocean  gyres  (Hurlburt  and  Hogan,  1998),  (5)  it  is  necessary  to 
resolve  small  islands  and  narrow  passages  which  affect  current  pathways  and  current  transports  in  many 
regions,  (6)  in  data  assimilative  mode  we  do  not  want  the  ocean  model  to  "fight"  the  data  because  the 
natural  behavior  of  the  ocean  model  is  inconsistent  with  the  observations,  and  (7)  a  very  high  horizontal 
resolution  model  is  needed  to  help  get  an  accurate  mean  sea  surface  height  field  to  add  to  the  deviations 
obtained  from  satellite  altimetry  (observations  alone  do  not  provide  sufficient  resolution  to  do  this). 

How  much  resolution  is  required  to  accomplish  these  goals?  Do  we  see  evidence  of  solution 
convergence  as  the  resolution  is  increased?  Sea  of  Japan  simulations  (Hogan  and  Hurlburt,  1997)  show 
major  impact  from  increasing  the  resolution  up  to  1/32°,  but  relatively  modest  changes  with  a  further 
increase  to  1/64°.  However,  a  Sea  of  Japan  model  cannot  be  used  to  address  issues  like  (4)  and  (5) 
above.  These  are  addressed  by  the  1/64°  Atlantic  simulation  running  as  part  of  this  DoD  Challenge 
project,  a  region  where  again  we  have  already  seen  major  impact  from  resolution  increases  up  to  1/32° 
(Hurlburt  and  Hogan,  1998). 


Results  from  FY97  and  FY98-00  DoD  HPC  Challenge  Projects  to  date 

A  1/16°  global  ocean  model  was  run  on  the  CEWES  T3E  under  an  FY97  DoD  HPC  Challenge  project. 
Before  extension  at  1/16°  resolution,  it  was  spun  up  from  rest  to  statistical  equilibrium  at  progressively 


higher  resolutions  (1/2®,  1/4®  and  1/8® )  outside  DoD  Challenge  forced  by  monthly  climatological  winds. 
It  was  continued  to  near  equilibrium  at  1/16®  resolution  with  the  climatological  forcing.  Then  two 
"identical  twin"  simulations,  initialized  at  two  different  times  from  the  climatological  simulation,  were 
run  1979-1996  forced  by  6  hourly  reanalysis  winds  (1979-93)  and  archived  operational  winds  (1994-96) 
from  the  European  Centre  for  Medium-range  Weather  Forecasts  (ECMWF).  The  first  simulation  was 
run  under  the  DoD  Challenge  project,  the  second  under  early  access  on  the  NAVO  T3E.  The  simulations 
showed  many  of  the  benefits  of  increased  resolution  outlined  earlier,  including  greater  eastward 
penetration  of  inertial  jets,  sharper  definition  of  oceanic  fronts,  and  better  simulation  of  meandering 
currents  and  the  mesoscale  eddy  field.  The  identical  twin  interannual  simulations  help  distinguish 
between  deterministic  and  nondeterministic  model  responses  to  the  atmospheric  forcing  and  are  also 
valuable  in  model-data  comparison  studies  and  dynamical  studies  of  oceanic  anomalies.  So  far  these 
results  have  been  used  in  publications  (1)  about  the  impact  of  model  resolution  on  the  global 
thermohaline  circulation  (Shriver  and  Hurlburt,  1998),  (2)  showing  that  El  Nino-generated  Kelvin  waves 
can  propagate  at  least  as  far  as  Petropovlovsk,  Russia  on  the  Kamchatka  peninsula  (Fig.  1)  (Metzger  et 
al.,  1998)  and  (3)  showing  effects  on  Gulf  Stream  variability  including  (a)  interannual  vs  climatological 
wind  forcing  for  the  model  and  (b)  the  impact  of  Atlantic  model  boundaries  at  9°N  and  51®N  by 
comparison  with  global  model  results  (Hurlburt  and  Hogan,  1998).  In  addition  these  model  results  have 
been  used  in  a  DoD  HPC  success  story  (Hurlburt  et  al.,  1998)  and  in  numerous  presentations.  Subsets  of 
model  output  have  been  distributed  to  researchers  outside  NRL. 

Results  from  the  1/32®  Pacific  model  and  the  1/64®  Atlantic  model  are  shown  in  figures  2  and  3.  The 
Pacific  model  provides  a  striking  example  of  oceanic  fronts  and  inertial  jets  (narrow  parallel  ribbons  of 
color)  that  can  span  a  large  ocean  basin  in  a  high  resolution  model  (as  found  in  observations).  Work  has 
also  begun  on  assimilation  of  satellite  altimeter  data  into  a  1/16®  version  of  the  Pacific  model.  In 
comparison  with  linear  solutions  and  coarser  resolution  nonlinear  simulations  (not  shown),  the  1/64° 
Atlantic  model  dramatically  demonstrates  the  impact  of  such  high  resolution  on  (a)  the  Gulf  Stream 
pathway  (marked  by  narrow  ribbons  of  color)  between  Cape  Hatteras,  North  Carolina  and  the  Grand 
Banks  off  Newfoundland  and  (b)  the  formation  of  a  large  nonlinear  recirculation  gyre  on  the  south  side 
of  the  Gulf  Stream  which  substantially  impacts  the  large  scale  shape  of  the  subtropical  gyre  (within  the 
area  surrounded  by  the  yellow  band).  Results  from  this  model  are  discussed  by  Hurlburt  and  Hogan 
(1998)  in  a  paper  on  the  impact  of  increased  resolution  on  Atlantic  simulations,  especially  on  the  Gulf 
Stream  system.  Results  from  both  FY98  DoD  Challenge  simulations  have  been  used  in  presentations. 


Figure  1.  Sea  level  (in  cm)  vs  time  at  Petropovlovsk,  Russia  (marked  by  a  red  dot  at  53.1°N,  158.4®E  on 
figure  2)  from  observed  tide  gauge  data  (solid  line)  and  a  1/16®  global  ocean  simulation  (dashed  line). 
The  simulation  was  run  as  part  of  an  FY97  DoD  HPC  Challenge  project  using  the  NRL  Layered  Ocean 


Model  (NLOM).  The  correlation  coefficient  between  the  two  time  series  is  0.87.  A  12-month  running 
mean  filter  was  applied  to  each  time  series  to  highlight  interannual  variability.  This  variability  includes 
sea  level  rises  due  to  the  1982-83, 1986-87  and  1991-92  El  Ninos  via  El  Nino  generated  Kelvin  waves 
which  first  traveled  eastward  along  the  equator,  then  counterclockwise  around  the  rim  of  the  North 
Pacific  to  this  distant  location.  These  Kelvin  waves  are  clearly  visible  in  animations  of  the  model 
simulations.  The  unobserved  sea  level  drop  between  1993  and  1994  in  the  model  time  series  coincides 
with  the  change  from  reanalysis  to  archived  operational  ECMWF  wind  forcing  for  the  ocean  model.  The 
tide  gauge  data  were  obtained  from  the  Integrated  Global  Ocean  Services  System. 


Figure  2.  Sea  surface  height  (SSH)  snapshot  from  the  1/32“  Pacific  version  of  NLOM  currently 
running  as  part  of  this  FY98-00  DoD  HPC  Challenge  project. 


Figure  3.  Sea  surface  height  snapshot  from  the  1/64°  Atlantic  version  of  NLOM  currently  running 

as  part  of  this  FY98-00  DoD  HPC  Challenge  project. 
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